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ABSTRACT 



A formative committee conducted a developmental study to 
produce a program evaluation instrument (PEI) that would be effective in 
conducting an adult basic education (ABE) program evaluation study at Nicolet 
Area Technical College ( NATC ) . Five procedures were used to complete this 
product. First, a literature review was conducted that assisted in the 
development of the product validation criteria. A review of benchmark studies 
for. program effectiveness provided direction in the evaluation instrument 
completion. Second, basic education staff held a series of investigatory 
meetings at which demographic trends affecting the ABE program were outlined. 
The information obtained that was used to develop the validation criteria for 
the PEI. Third, the formative committee refined the validation criteria, 
which included monitoring and screening data, indicators of program quality 
in ABE, and development of the PEI. Fourth, all criteria considered critical 
in an effective PEI were measured as a standard against the developed PEI. 

The PEI was determined to be appropriate for the intended use. Fifth, the PEI 
was presented to the summative committee, which validated that the PEI met 
required criteria and was appropriate for the ABE program's evaluation 
process. (Appendixes include 12 references, indicators of program quality, 
monitoring and screening criteria and standards, and the PEI.) (YLB) 
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DEVELOPMENT OF A PROGRAM EVALUATION INSTRUMENT FOR 
THE ADULT BASIC EDUCATION PROGRAM AT 
NICOLET AREA TECHNICAL COLLEGE 
By 

Luis C. Karl 
April, 2000 



The problem in this developmental study was that the Adult 
Basic Education (ABE) program at Nicolet Area Technical College 
needed to conduct a program evaluation but lacked an appropriate 
evaluation instrument. The ABE department needed to develop a 
program evaluation instrument that would satisfy federal, state, 
and institutional requirements for effective program delivery. 
The purpose of the study was to apply appropriate monitoring and 
screening criteria and standards to the development of an ABE 
program evaluation instrument. 

There were two research questions for this study. First, 
"What is the format of a program evaluation instrument that will 
provide the criteria necessary to effectively evaluate the ABE 
program's instructional delivery?" Second, "How will the 
evaluation instrument integrate with strategic plans concerning 
assessment, instructional long range planning, and data scanning 
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with links to new components of program development?" A 
developmental study was conducted to create a program evaluation 
instrument to analyze data that enabled judgments to be made 
about the ABE program's effectiveness. 

The study resulted in the development of a program 
evaluation instrument that integrated the strategic plan data 
elements with the monitoring and screening criteria and standards 
established for effective program assessment and planning. It 
was concluded that an effective program evaluation instrument 
should contain matrices that document evidence that the program 
accomplishes its purpose by addressing student success through 
recruitment, access, cost, retention, curriculum, and employment 
preparation. It was recommended that effective program 
evaluation instruments contain monitoring and screening criteria, 
prioritized recommendations, and program component implementation 
strategic plans. 
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Chapter 1 
INTRODUCTION 
Nature of the Problem 

The Wisconsin Technical College System Board (WTCSB) 
developed a new paradigm for the delivery of basic education in 
Wisconsin in response to a rapidly changing technology 
environment. Rapid change is coupled with increased costs for 
education and decreasing funding sources. Nicolet College 
directed the ABE program to investigate its own effectiveness and 
make recommendations for program improvement to meet current and 
near- future customer needs. The problem is that a program 
evaluation instrument needed to be developed to assist in the 
process of investigating the basic education program' s 
effectiveness (Wisconsin Technical College System 1994). 

Purpose of the Study 

The purpose of this study was to gather information that 
provided benchmark recommendations and WTCSB -supported guidelines 
to be used in the development of an evaluation instrument for the 
Adult Basic Education (ABE) program at Nicolet Area Technical 
College (NATC) . Past practice of NATC program delivery did not 
apply to future customer needs, which needed to be researched. 

An appropriate program evaluation instrument would be used to 
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determine if program delivery meets current and near-future 
customer demands. 

Research Questions 

There were two research questions for this study. First, 
"What is the format of a program evaluation instrument that will 
provide the criteria necessary to effectively evaluate the Adult 
Basic Education program's instructional delivery?" Second, "How 
will the evaluation instrument integrate with strategic plans 
concerning assessment, instructional long range planning, and 
data scanning with links to new components of program 
development ? " 

Definition of Terms 

Adult Basic Education (ABE) . Adult Basic Education is 
instruction offered to adults on a continuum, beginning with 
elementary levels and culminating with competencies equivalent 
with grade 12.9 (graduation equivalent). 

Alternative high school program (AHSP) . Wisconsin statute 
118.15 outlines a secondary school with a nontraditional 
curriculum that is usually administered through the district 
vocational-technical or community college. 

Computer software programs for ABE . Such courses are 
designed to provide orientation to existing software programs, 
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including word processing, that are used in basic skills 
education. This course provides students with essential exposure 
to computer technology (now a recognized basic skill) and access 
to excellent multimedia instructional resources in basic 
education. 

Full-time Equivalent (FTE) . Full-time equivalent is a 
formula used to calculate the cost of instructing students 
through the accumulation of one course credit . Thirty course 
credits equal one FTE. The institution counts the unduplicated 
number of enrolled students during a fiscal year (July 1 through 
June 30) that generate course credits. The cost of instructor 
salary, divided by the number of FTEs generated may be used to 
determine a cost per student. 

Instructional Services . Instructional services refers to 
administrators responsible for academic programs at the college. 

Instructional Technology . Instructional technology is the 
theory and practice of design, development, utilization, 
management, and evaluation of processes and resources for 
learning . 

Retention . Retention is the ability to keep; as to keep 
students enrolled in ABE programs until graduation or course/ 
competency completion. Retention includes transition from basic 
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education programs to post -secondary programs leading to an 
associate degree or a competency certificate. 

Review . Review refers to a process of investigating an area 
of the Basic Education program that has continual decrease in 
enrollment or FTEs over an established period of time. 
Recommendations are made, as part of the review, for program 
component improvement to increase enrollment or to eliminate the 
program component through reallocation of resources. 
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Chapter 2 

REVIEW OF THE LITERATURE 

Changes in the economy in the last 25 years have been 
disastrous for those who left school before receiving a high 
school diploma. Today, a 30 -year-old male who has dropped out of 
high school earns one-third less than a comparable worker without 
a diploma would have earned in 1971. Adult basic education (ABE) 
is a second chance educational route for those adults wishing to 
improve their lives. Basic education (B.E.) programs need to 
focus on current employability demands, in addition to a basic 
high school equivalency certificate that will improve the lives 
of their students who complete ABE programs (Murnane, Willet, & 
Tyler, 1998) . 

Present Status of Topic 

The Wisconsin Technical System Board (WTCSB) presented A 
Strategic Policy Recommendations Report (June, 1994) to the 
Wisconsin Technical College System institutions that called for a 
shift to a new education paradigm that will take basic skills 
education into the 21st century. This shift is organizationally 
embodied in an adult school-to-work continuum which serves as the 
framework for basic skills education programming. 

With the call for new andragogy and terminology to replace 
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outdated perception of ABE programs, the WTCSB shifted to a 
curriculum which is student -centered, driven by content knowledge 
teaching, which is socially and culturally relevant, and which 
develops critical thinking, life-long learning skills, and 
English language proficiency (WTCSB, 1994) . The programs are now 
called Basic Education (B.E.). 

Research in the development of the "New Paradigm" for basic 
skills in Wisconsin was based on increasing movements to consider 
student outcomes as a priority. This process could be 
accomplished by including activities that perform extensive 
student assessment through improved student assessment models, 
and to refocus institutional missions from serving the 
institutional needs to serving students in their learning. 
Assessment has been identified as a prominent driver for 
institutional and program strategic planning. 

Further, the new paradigm was proposed to consider the 
apparent conflict that may exist in the belief that the primary 
goal of education is student achievement, while current higher 
education institutions may function according to other 
influences, such as maintaining employment of long-term 
instructors. Eventually, the term, "learning paradigm" should be 
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less of a shift in the approach to education to more development 
of systems that focus on student learning than instruction. 

The reason for the paradigm shift in Wisconsin's vocational 
and technical college instructional delivery was based on meeting 
student needs. These needs were outlined in a document that 
listed nine indicators of program quality in adult basic 
education programs in Wisconsin that address learner success 
(Wisconsin Technical College System Board, 1998) . 

Recent changes in the method of program delivery at NATC 
occurred because of rapid technology development and the sudden 
need of adult learners to utilize computers in their daily lives. 
The adult basic education program must respond to stakeholders by 
providing services to learners that meet their immediate needs. 
The WTCSB called upon all technical college ABE programs to 
provide services in computer literacy to all their enrolled 
clients (WTCSB, 1997) . 

All WTCS learning centers have been directed to conduct 
program evaluations that will identify strengths, weaknesses, 
opportunities, and threats in their ABE programs that: (a) assist 

in the enrollment and completion of occupational programs 
and/other post secondary education, (b) attain and retain 
meaningful and sustainable employment, (c) effectively support 
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academic success within the family, (d) become informed and 
productive participants in the civic life of the community, and 
(e) become self -directed and life-long learners. To achieve 
these purposes, the WTCSB set policy and guidance for the 
effective implementation of its B.E. programs to establish 
instructional bridges and organizational structures which help 
move learners up the educational continuum into occupational and 
post secondary programs (NATC, 1997) . 

With the continuum of educational services concept, in adult 
basic education (ABE) , was the recognition that basic skills 
education is much broader in scope than generally perceived. 

Adult basic education programs and services range from those 
needed by adult learners at the lowest level of literacy to 
adults seeking a high school credential, to the high school 
graduate needing a high school course as a prerequisite for entry 
into a technical college program, to the student enrolled in a 
technical college program who needed supplemental assistance to 
ensure program graduation (Beder, 1998) . 

Research by Staff 

Classical studies in teacher-as-researcher are part of 
recent literature on educational evaluation and reform. Action 
research by higher education staff has provided a team approach 
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and a forum for sharing questions, concerns, and results. 

Teacher research in program planning has promoted professional 
dialogue and has created a more professional culture in schools. 
The concept of action research can be traced back to the early 
works of John Dewey in the 1920s and Kurt Lewin in the 1940s. 

The concept of research by all members of the institution, 
including teachers, is recommended as a human resources component 
of the organizational frames described by Bolman and Deal (1997) . 
Because the practitioners (teachers) are closer to the clients 
and the immediate problems associated with program access and 
student retention, membership by instructional staff should be 
included in program model design (Bolman & Deal, 1997) . 

Benchmark Studies 

Terry O'Banion (1997) , in his recommendations for program 
changes, recommends to shift perspectives in higher education 
from instruction to student learning. Those topics are judgement 
of institutional success on the quality of student learning and 
shared responsibility in student learning between the college and 
the student (O'Banion, 1997). 

The Educational Resources Information Center (ERIC) has 
featured benchmark arguments in favor of rapid change in the way 
institutions of higher education provide services to students 
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that focus on learning, rather than instruction. The change that 
is needed to respond to customer demand can no longer be 
incremental. Rather, change must be system-wide and holistic. 
Learning must be placed first in every aspect of policy, mission, 
programs, and practices of colleges that hope to prepare its 
clients for the immediate future (Barr, & Tagg, 1995) . 

Technology 

Benchmark studies show that technology is rapidly changing 
the approach people use to provide educational opportunities. 
Students and staff must remain abreast of technology advancement 
in order to maintain their competitiveness in the job market. If 
community colleges fail to provide the opportunity to learn these 
skills they risk a decrease in student enrollment, as students 
will seek colleges better able to accommodate their needs. In 
developing a program evaluation instrument, colleges need to 
consider the impact of computer technology, both from a cost and 
benefits perspective, in providing quality access to technology. 
Program evaluation instruments should include categories for 
technology-budget considerations, so that necessary services to 
students can be maintained. 

Doucette (1994) describes two ways technology is 
implemented. The first type of implementation is the use of 
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technology as a simple add-on to enhance current instructional 
methods. With multimedia technology, instructors can create 
attention- catching lectures and can also generate plans which 
allow them to change format based on student understanding and 
interest. The second type of implementation Doucette (1994) 
describes is the more complex process of using technology to 
transform both the teaching and learning functions. Systems that 
provide students with access to a multimedia computer station 
equipped with programs of course content allow them to work at 
their own pace. Students can focus on individual areas of 
weakness because the course content comes from computer disks 
rather than from formal lectures. Benefits to consider in the 
use of multimedia technology, in program planning and evaluation 
is the student receives immediate feedback, learning style 
accommodation, improved retention and understanding of the 
material, increased levels of student participation and interest, 
and more opportunities for team learning (Doucette, 1994). 

Using computer-based technology in a lab-type setting can 
provide flexibility in instructional delivery for students who 
have difficult schedules, such as adults with short study 
periods. The Flex Lab at Santa Fe Community College offers 
courses which are accessed by computer so that students can work 
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on assignments at their own pace in a relaxed, non-competitive 
learning environment. The lab is staffed by trained personnel 
who can answer questions and provide guidance to the students . 
Lab- type instructional programming can also increase a college's 
potential to provide distance education if programs can be 
accessed through the Internet (Ortego & Richards, 1995) . 

Technological advancement has established a significance in 
ABE programs to the extent that basic computing skills are 
considered a fourth component of basic skills (reading, writing, 
arithmetic, and personal computer use) . The WTCSB has called on 
system institutions to develop policy that makes personal 
computer technology, in B.E. programs, a priority in funding and 
access for students. 

Programs that effectively use technology have a carefully 
designed technology plan that is a part of the overall program 
improvement plan. To realize the benefits of technology, 
programs must develop a plan for integrating technology into the 
curriculum. An effective technology plan is based on the shared 
vision of educators, students, community members, and leaders 
with technological expertise. Technology is transforming society 
and the ABE program does not have a choice as to whether 
technology will be incorporated, but how it will be used to 
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enhance learning (North Central Regional Educational Laboratory & 
Illinois State Board of Education, 1995) . 

Integral to the reforms of basic education (B . E . ) is the 
coupling of curriculum development with emphasis on the 
importance of academic knowledge in technical training. The 
changes in B.E. have prompted studies of the Rindge School of 
Technical Arts (RSTA) in Cambridge, Massachusetts, as a model 
practice by organizations such as the National Center for 
Research in Vocational Education, University of Wisconsin- 
Whitewater's Center for Work and Education and Boston's Jobs for 
the Future. The Rindge School conducted a program evaluation to 
plan for the elimination of the divisions between hand and mind 
and now requires rigorous academics with technical training for 
all vocational students. The problem is that emphasis on 
academics may prevent students from finding adequate time to 
learn the trades that will provide them with their future income 
(Dai Vo, 1996) . 

The mission of upgrading the curriculum and program 
offerings for vocational-technical and community college students 
is to emphasize the tendency of technology to create quality 
training for the academically disadvantaged. Curriculum that 
includes educational technology should provide greater access and 
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quality of education to disadvantaged individuals by reducing the 
cost of access to the best instruction. New technologies, if 
properly implemented, should decrease the discrepancy between the 
information haves and have-nots (Snider, 1996) . 

Education in the ABE programs should be individualized to 
the needs of each student. Enrollment should be open-entry, 
open-exit, with a variety of schedules to accommodate students' 
personal lives. Education should involve more holistic, 
knowledge -based intellectual frameworks, rather than "short-lived 
retention of fractionated contextual cues" (Barr & Tagg, 1995) . 

Educational climate should foster the belief that student 
learning is the central objective of all employees in the 
institution. The mission of the college must reflect that belief 
and internal and external stakeholders should carry this 
philosophy with them in their daily work, regardless of their 
assignment . 

Summary 

It is the mission of the B.E. programs to facilitate the 
acquisition by adult learners of basic academic, research, 
critical thinking and technology skills and culturally and 
socially relevant social science, mathematical and scientific 
knowledge. All Wisconsin B.E. programs must facilitate the 
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acquisition of college survival skills, a global understanding of 
careers, and understanding of the employment market and work. 
Further, ABE programs should develop and facilitate educational 
programs that collaborate with other educating entities, public 
and private human service agencies, business/industry and labor, 
and respond to technological advancement. 

Institutions need to conduct evaluations on their programs' 
effectiveness in meeting current and future customer needs. 
Program evaluation components need to include categories that 
address the critical issues discussed by benchmark studies, 
trends, staff research/recommendations, and customer needs in 
higher education. 
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Chapter 3 

METHODOLOGY AND PROCEDURES 
Problem Solving Methodology 

The problem solving methodology that was used in this study 
was development. Development was used through an organized 
procedure to produce a product which was used to analyze data 
that enabled judgments to be made about the Adult Basic 
Education (ABE) program's effectiveness in meeting the strategic 
plan goals and in the subsequent improvement of the ABE program. 

Procedures 
Data Gathering 

Five procedures were used to complete this development 
practicum. First, a review of literature concerning basic 
education program evaluations was conducted. The review included 
directives from the Wisconsin Technical College System (WTCSB) 
that addressed the indicators of program quality in Wisconsin in 
Adult Basic Education (Appendix A) , Nicolet Area Technical 
College (NATC) , and benchmark studies of program evaluation 
processes conducted within the past five years. 

Second, a self-study on the ABE program at NATC was 
conducted during the fall semester, 1998, by the basic education 
staff. The study began with a discussion by the basic education 
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staff, who identified trends that should be considered as 
evaluation criteria to be used during the program evaluation 
process. This discussion began during a full-day planning 
meeting on October 2, 1998. Among the topics for consideration 
by the discussion group were those outlined by authorities in 
higher education in their recommendations for program changes to 
shift perspectives in higher education from instruction to 
student learning. Those topics were judgement of institutional 
success on the quality of student learning and shared 
responsibility in student learning between the college and the 
student . 

Further discussion centered on the concept of creating a 
seamless system of delivery that would provide access to 
educational services for learners as they need them, when they 
need them, and wherever they need them. This component of the 
discussion clarified the importance of the outreach centers as 
part of the ABE program. Seven NATC outreach centers provide 
access and quality of ABE services to remote areas of the 
district for students who are unable to travel the distance to 
the main campus. A significant component of the discussion 
centered on the continual identification, development, testing, 
implementation, and assessment of a range of effective learning 
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technologies including new applications of computer and 
information technology for ABE students. Budgeting for new 
equipment, facilities, and staff were important sub -components of 
the development of a program evaluation instrument tool . 

Consideration of faculty work load, work site, and job 
description were included in the development process of program 
evaluation. For example, faculty whose primary responsibility is 
the design of learning methods and environments, with less 
emphasis on the traditional responsibility of instruction, must 
be considered; cross-disciplinary teams of staff who work 
collaboratively to devise programs to increase student 
competencies and retention should be available on campus and 
outreach centers. The diversity of ABE work- related environments 
in the Nicolet College district required extensive consideration. 

Philosophy of program delivery was also discussed. 

Education in the ABE programs should be individualized to the 
needs of each student. Enrollment should be open-entry, open- 
exit, with a variety of schedules to accommodate students' 
personal lives. Education should involve more holistic, 
knowledge -based, intellectual frameworks. Educational climate 
should foster the belief that student learning is the central 
objective of all employees in the institution. The mission of 
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the college must reflect that belief and internal and external 
stakeholders should carry this philosophy with them in their 
daily work, regardless of their assignment. 

A list of trends that impact the ABE program delivery were 
developed to assist in the establishment of the criteria. The 
trends included, but were not limited to, the following: (a) 

Wisconsin's Work, Not Welfare (W-2) law, which influenced a 
decrease in ABE enrollments during the 1997-98 academic year and 
established the need for more flexible scheduling to meet 
employment schedules of part-time students; (b) an increased 
enrollment of alternative high school students in ABE programs; 

(c) increased requests for accommodations for students with 
disabilities; (d) technology- related learning has increased hours 
that most students spend in the learning centers; (e) computer 
literacy designated as the "fourth" basic skill in ABE program 
delivery; (f) increased demand for customized learning 
certificate programs for employment; (g) increased difficulty in 
basic skills-related learning to satisfy workplace education 
skills analysis recommendations; (h) increased competition for 
funding through grant sources, which has been historically 
reserved for institutions of higher learning; (i) adult basic 
education partnering with business and industry, as outlined in 
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the WTCSB guidelines; (j) competition from outside instructional 
resources to serve local clients; (k) a shift in focus from 
serving lower level students to serving higher academic needs in 
basic skills, to be eligible for grant funding; and (1) a top 
priority in the college strategic plan to increase services to 
Native Americans in their communities, local learning centers, 
and Nicolet College district campus programs. 

Criteria 

Formative Committee 

Information from the Teaching, Curriculum, and Instructional 
Resources (TCIR) college standing committee was reviewed in 
relation to recommended components for inclusion in the program 
evaluation instrument that was developed and refined by a 
formative committee, which was established through the basic 
education staff and college staff who work closely with 
components of the basic education program on the main campus. 

The formative committee consisted of four full-time faculty in 
the ABE program, two members of the ABE Advisory Committee, and 
two NATC staff members outside of the ABE program. See Appendix 
B for a rationale for membership selection in the formative 
committee. Subsequent planning meetings were held by the 
formative committee, during February and March, 1999, to 
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establish the criteria that was used to develop the program 
evaluation instrument. Through the research and discussion 
process, the criteria was refined into short statements and 
established for the program evaluation instrument by the 
formative committee (Appendix C) . 

Third, the formative committee gathered data from the 
institutional data processing department that provided a history 
of full-time equivalent (FTE) student enrollment in each of the 
ABE program areas studied in the program evaluation. The 
formative committee identified recommendations, insights, and 
comments about any aspect of the educational offerings or 
programs during the evaluation process that would be integrated 
into the evaluation instrument components. The Monitoring and 
Screening Criteria and Standards were listed as criterions one 
through six. Trends that indicated continual changes in 
enrollment in certain areas provided data to recommend review. 
Percentages of decreasing enrollment, throughout an extended 
period, may be established as a check point for review. For 
example, a decrease of 10% in enrollment in a specific area of 
learning may be an alert to review the area in a program 
evaluation exercise to recommend change or improvement. 
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Formative committee meetings were held during the fourth 
week of February, and the first week of March, 1999, where 
committee members discussed and compiled information in each of 
the criteria categories listed, which were eventually written 
into the evaluation instrument's two components (Appendixes C and 
D) . One component was titled, Monitoring and Screening Criteria 
and Standards (Appendix C) , with columns that listed data 
elements, definition, measure, benchmark, and review level. The 
second component was titled Program Evaluation and Improvement 
Recommendation Form (Appendix D) , which had three columns that 
addressed the following: (a) evaluation questions and topics for 

consideration, (b) analysis, insights, comments, and (c) action. 

The ABE program evaluation formative committee established 
data elements, from the trends study, that responded to the 
Nicolet College institutional and ABE department goals and plans. 
These data elements were listed under the appropriate criterion 
in the instrument. A definition and measure for each data 
element was established, with a third section for action 
recommended if a review of the program process was inadequate, or 
if success was not at acceptable levels. Trends in enrollment 
were considered to be important monitoring and screening criteria 
for measuring program effectiveness. This information responded 
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to the trends criteria established for the program evaluation 
study (Appendix C) . 

Summative Committee 

Fourth, the formative committee presented the program 
evaluation instrument to the summative committee on March 15, 
1999. The summative committee consisted of the ABE cluster 
coordinator, two ABE Advisory Committee members, the vice- 
president of instruction, and a member of the NATC Teaching, 
Curriculum, and Instructional Resources (TCIR) standing committee 
(Appendix E) . 

Validation of the Criteria 

The summative committee validated the criteria established 
for the program evaluation instrument by comparing the following 
criteria components to the evaluation instrument to verify if 
those components were included: (a) The instrument must contain 

data elements that address ABE services that respond to the nine 
indicators of program quality (Appendix A) ; (b) the institutional 

strategic plan goals must be incorporated into the basic 
education strategic plan and goals (Appendix F) , which must be 
tied to the WTCSB regulations and guidelines for the operation of 
adult basic education programs; (c) each criterion must include 
data elements of program goals (Appendix C) , including current 
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effectiveness and efficiency that supports program success and 
review levels; and (d) trends that may be impacting the strategic 
goals and plans for program success (Appendix D) . 

Development of the Product 

Fifth, the formative committee compiled all evaluation 
topics and data gathered for consideration on a draft evaluation 
instrument form. The process of entering the criteria on a draft 
evaluation form was done during a series of meetings that were 
task-oriented sessions that focused on constructing the document 
in a format that was in a matrix configuration. The product was 
two components that outlined the monitoring and screening 
criteria and standards crucial to successful program operation 
and a section for program improvement recommendations. It 
contained evaluation questions and topics that addressed 
strategic goals of the ABE program, including performance 
standards and measures used to evaluate program effectiveness 
(Appendixes C and D) . This instrument was the draft model of the 
program evaluation instrument. The instrument construction 
sessions were completed by March 22, 1999. These work meetings 
resulted in the actual development of the program evaluation 
instrument . 
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Validation of the Product 



The summative committee then validated if the product was a 
useful tool for an effective program evaluation. The validation 
process was conducted by comparing the established criteria for 
program effectiveness against the components of the new program 
evaluation instrument. The Monitoring and Screening Criteria and 
Standards were listed on the instrument as criterion one through 
six. 

The ABE program evaluation formative committee established 
data elements that responded to the institutional and department 
goals and plans. These data elements were listed under the 
appropriate criterion in the instrument. A definition and 
measure for each data element was established, with a third 
section for action recommended if a review of the program process 
was inadequate, or if success was not at acceptable levels. 

The purpose of this validation exercise was to determine if the 
critical attributes of a program evaluation instrument were in 
place, according to the established criteria. The summative 
committee completed its comparative study by March 29, 1999. 

Assumptions 

For this study, it was assumed that the members of the 
formative committee were able to identify appropriate trends that 
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should be considered in planning for an effective program 
evaluation instrument and a subsequent program evaluation. It 
was assumed that the diversity of the Nicolet district basic 
education staff responsibilities could be considered in the 
components of the instrument. 

It was also assumed that the recommendations made by the 
summative committee are beneficial to addressing current and 
future trends in ABE education. Finally, it was assumed that 
recommendations are reasonable and could be implemented without 
financial prohibition. 

Limitations 

The data collected by the trends committee may not be 
appropriate to future needs of potential clients. Financial 
restrictions may limit the extent to which implementation of 
recommendations can be made. The basic education instructional 
duties and learning center environments are diverse, and 
therefore challenge the extent to which an effective program 
evaluation instrument could address all of the program unique 
conditions. Monitoring and screening data that was used to 
develop the instrument may only apply to the current year of 
program evaluation. 
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Chapter 4 
RESULTS 

A developmental study was conducted by a formative committee 
to produce a program evaluation instrument that would be 
effective in conducting an adult basic education (ABE) program 
evaluation study at Nicolet Area Technical College (NATC) . Five 
procedures were used to complete this product . 

First, an extensive literature review was conducted, which 
assisted in the development of the product validation criteria. 

A review of benchmark studies for program effectiveness was 
conducted to provide direction in the evaluation instrument 
completion. Information from the Teaching, Curriculum, and 
Instructional Resources (TCIR) college standing committee was 
reviewed in relation to recommended components for inclusion in a 
program evaluation instrument . 

Second, a series of investigatory meetings were held by 
basic education staff, where demographic trends affecting the ABE 
program were outlined. These meetings resulted in an 
accumulation of information that was used to develop the 
validation criteria for the program evaluation instrument 
(Appendix C) . 
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Third, the formative committee (Appendix B) refined the 
validation criteria, which included monitoring and screening 
data, indicators of program quality in adult basic education, and 
the development of the program evaluation instrument (Appendixes 
C and D) . The instrument was organized to establish three column 
categories that would respond to evaluation questions and topics 
regarding the successful operation of a basic education program. 
The topics and data components were the criteria standards 
against which the product was measured. 

The three columns were evaluation questions and topics for 
consideration; analysis, insights, and comments; and, action 
indicated for success. Six categories were established to apply 
to the three columns for evaluation: (a) progress in current 

program, (b) current effectiveness and efficiency (include 
multiple options for access) , (c) trends affecting the Basic 

Education program, (d) assessment of the program system, (e) 
instructional technology, and (f) prioritized recommendations for 
improvement . 

The formative committee also considered district economic 
development needs, technology program needs on campus, and 
infusion of basic education with other programs on campus . The 
criteria included: (a) monitoring and screening criteria and 
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standards that were provided by the Teaching, Curriculum, and 
Instructional Resources (TCIR) standing committee; (b) evaluation 
questions and topics for consideration, including current 
effectiveness and efficiency of ABE program, Indicators of 
Program Quality (Appendix A) , and multiple access options 
available to clients; (c) trends affecting the program or area; 

(d) consideration of assessment loops currently being used in the 
ABE program; (e) consideration of instructional technology being 
used and what should be considered for future use; and (f) 
student data reports on enrollment and course completion from 
previous years . 

Fourth, all of the criteria considered to be critical in an 
effective program evaluation instrument was measured as a 
standard against the developed program evaluation instrument. 

The validation component was conducted by the summative committee 
(Appendix E) . The summative committee compared the ABE 
monitoring and screening standards in the program evaluation 
instrument to the established criteria and standards for the 
college. The result was that the summative committee determined 
that the instrument was considered to be appropriate for the 
intended use, based on the criteria from which it was measured. 
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The two research questions were answered through the 
development of a program evaluation and improvement 
recommendation form, as follows: The format of the product 

included categories that addressed evaluation questions and the 
following topics for consideration; analysis, insights, and 
comments; and indicated action to complete the evaluation. 
Monitoring and screening criteria and standards, including data 
elements, definition/measures of the data elements, benchmarks, 
and review levels for insufficient progress, were compiled to 
guide the program evaluation instrument. 

Specific categories in the program evaluation instrument 
format included (a) current program effectiveness and efficiency, 
(b) verification of multiple access options for students to 
appropriate instruction, (c) trends affecting basic education, 

(d) assessment loops utilized through collaboration with internal 
and external offerings, (e) instructional technology review and 
two-year plan, and (f) prioritized recommendations for program 
improvement . 

Five, the program evaluation instrument was presented to the 
summative committee, during a scheduled meeting on March 15, 

1999. The summative committee met again on March 22, 1999. That 
meeting involved an item analysis of the program evaluation 
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instrument, against the established criteria. The summative 
committee subsequently validated that the program evaluation 
instrument met the required criteria and was appropriate for the 
adult basic education program's evaluation process. 

The form was further enhanced through the inclusion of a 
budget recommendations form for program improvement . The budget 
form included categories for capital cost, staff cost, base 
allocation increase, technology improvement, and/or physical 
resources improvement or modification. The result of the program 
evaluation instrument development effort was a product that can 
be used effectively in accordance with the research questions. 




38 



38 



Chapter 5 

DISCUSSION, CONCLUSIONS, IMPLICATIONS, AND RECOMMENDATIONS 

Discussion 

Nicolet Area Technical College's (NATC) Basic Education 
(ABE) program needed to develop a program evaluation instrument 
that would be effective in assessing the program's effectiveness. 
Changes in the adult basic education paradigm, coupled with rapid 
advances in technology prompted the ABE program to review 
benchmarks in ABE delivery, recognize state and local policy, and 
make recommendations for program delivery. 

Recent studies have shown a definite change in program 
delivery methods that focus on student learning, rather than 
instruction. Literature reviews have strongly favored program 
changes to respond to customer need, rather than traditional 
course construction and delivery. 

Instruction must be cost efficient, include computer 
literacy training, provide flexible scheduling, and have value 
toward application of skills learned. Literature has further 
indicated the need for change to be system-wide and holistic, 
rather than incremental. Institutional program delivery change 
must be student, rather than staff, centered. 
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Benchmark studies have shown that program evaluation 
instruments must address the issues of program delivery by 
including a focus on curriculum updates and appropriateness. 
Assessment loops for program integration with other departments, 
business and industry, and post -secondary education should be a 
part of the evaluation measures. Focus must place priority on 
the student needs, regarding access to and value of programs. 
Program delivery should not be focused on the convenience and 
preference of the staff. Benchmark studies have shown that 
additional emphasis on program evaluations should be placed on 
new and expanding funding sources, technology plans, and 
professional development. 

Conclusions 

There were two research questions for this study. First, 
"What is the format of a program evaluation instrument that will 
provide the criteria necessary to effectively evaluate the ABE 
program's instructional delivery?" Second, "How will the 
evaluation instrument integrate with strategic plans concerning 
assessment, instructional long range planning, and data scanning 
with links to new components of program development?" 

The research questions for this study were answered. First, 
the format of an ABE program evaluation instrument that can 
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provide the criteria necessary to effectively evaluate the ABE 
program's instructional delivery included categories that 
addressed evaluation questions and topics that included the 
following: (a) a current report on progress of the ABE program, 

when considering recruitment, retention, and program/ goal 
completion, (b) current effectiveness and efficiency, (c) trends 
affecting the ABE program, (d) assessment loops that address 
required outcomes for program success, (e) instructional 
technology considerations, and (f) prioritized recommendations 
for improvement . 

Second, the analysis, insights, comments, and indicated 
action section were necessary to complete the evaluation 
components that addressed environmental scanning, trends, and 
future needs. Monitoring and screening criteria and standards, 
including data elements, definition/measures of the data 
elements, benchmarks, and review levels for insufficient 
progress, were compiled to guide the program evaluation 
instrument. Program evaluation instruments must address all 
aspects of program delivery that impact student success. 

Implications 

A new paradigm for the delivery of ABE in Wisconsin called 
for response to rapid change in demographics, technology, and 
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increased costs in serving adult basic education students. If 
institutions that deliver ABE services develop strategic plans to 
respond to the new paradigm, an effective program evaluation 
instrument can enhance current and future planning . 

A program evaluation instrument that can respond to the 
monitoring and screening criteria and standards for effective ABE 
program delivery, can enhance the efficiency of program 
evaluations and assist in program planning through the analysis, 
insights, comments and action areas indicated for change/ 
improvement. Annual program reviews can be linked to previous 
use of the developed instrument, through the establishment of one 
and two-year plans that overlap evaluations. Budget requests can 
be justified through sections of the program evaluation 
instrument that indicate need through analysis and measure 
against the criteria for effective program delivery. 

Recommendations 

Staff 

It was recommended that program staff be involved in the 
development of program evaluation instruments. The process of 
involving the basic education instructional staff in the research 
and development of a basic education program evaluation 
instrument has provided them with a clearer understanding of the 
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scope of program operation. An instrument that was effectively 
designed to measure what it purported to measure, required 
extensive research into the critical attributes and criteria for 
monitoring and screening data that comprise the components. 

Implementation 

It was recommended that Nicolet Area Technical College 
(NATC) consider the approval of implementing the developed 
program evaluation instrument to the required ABE program 
evaluation process for the current academic year. This approval 
would come from the Teaching, Curriculum, and Instructional 
Resources standing committee (TCIR) for immediate implementation. 

Instrument Adoption 

It was also recommended that the institution consider 
adopting the instrument as a model tool in various program 
evaluations. Adoption of the instrument could be accomplished 
through discussion of the instrument's initial use, as an example 
of effectiveness. Dissemination of the instrument could be 
conducted through the TCIR standing committee meetings, which 
monitor program change. 

An adaptation of the Basic Education program evaluation 
instrument could be completed through various program needs, by 
introducing the instrument to each college department and 
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providing guidance from TCIR in adapting it to specific program 
needs . 

Further Study 

It was further recommended that further study be conducted 
on the monitoring and screening criteria and standards for 
program evaluation contained in the instrument . Annual updates 
on the strategic plan component and trends effecting the ABE 
program evaluation are recommended. Indicators of program 
quality in basic education programs should be further studied to 
align the indicators with current environmental scanning data. A 
comparison of the indicators of quality with potential ABE 
clients in the district should provide data for predicted success 
in service to the people of northern Wisconsin. 
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Indicators 



INDICATOR 1 



INDICATOR 2 



INDICATOR 3 



INDICATOR 4 



INDICATOR 5 



INDICATOR 6 



INDICATOR 7 
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Appendix A 

of Program quality in Wisconsin Adult Basic Education 

: LEARNER PROGRESS IN EDUCATIONAL ATTAINMENT 

Learners demonstrate progress toward attainment of 
basic education competencies. 

: LEARNER ADVANCEMENT TOWARD GOALS 

Learners advance in the instructional program or 
complete the program educational requirements that 
allow them to pursue their educational, family, 
community, workplace, and personal goals. 

: STUDENT RETENTION 

Students remain in the program long enough to meet 
their educational needs/goals. 

: PROGRAM PLANNING PROGRESS 

Program has a planning process that is on-going 
and participatory, guided by evaluation and based 
on a written plan that considers community 
demographics, needs, resources, and economic and 
technological trends, and is implemented to its 
fullest extent. 

: CURRICULUM AND INSTRUCTION 

Program has curriculum and instruction that 
addresses a variety of student learning styles and 
levels of student needs. 

ON-GOING STAFF DEVELOPMENT PROCESS 
Program has an on-going staff development process 
that is responsive to the specific needs of its 
staff, offers training in the skills necessary to 
provide quality instruction, and emphasizes 
practice and systematic follow-up. 

STUDENT SUPPORT SERVICES 

Program identifies students' needs for support 
service and makes services available to students 
directly or through referral to other educational 
and service agencies. 
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INDICATOR 8 : 



INDICATOR 9: 
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COORDINATION OF SERVICES 

Program successfully coordinates services for the 
population in the community identified in the 

(appendix continues) 

Adult Education and Family Literacy Act as 
under- served regarding literacy and basic 
education. 

VOLUNTEER SERVICES 

Programs will coordinate services and demonstrate 
collaboration among technical college districts, 
community-based organizations, and volunteer 
literacy providers. 
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Appendix B 
Formative Committee 

Members of the Formative Committee were selected because of 
their affiliation with the B.E. program through their current 
assignments and the benefits that effective B.E. programs can 
provide for their students. 

♦ Four full-time faculty in the B.E. program were selected: 

(a) mathematics instructor, (b) communications/English 
instructor (c) reading skills instructor, and (d) basic 
computer software skills instructor. 

♦ Two members of the B.E. Advisory committee were selected: 

♦ The NATC Registrar served from a post -secondary programs 
perspective . 

♦ One academic advisor from the general education faculty 
served with the connection of assisting enrolled students 
with remediation and skills enhancement courses in the B.E. 
program. 

♦ Two instructors outside of the B.E. program consisted of an 
auto mechanics instructor and a nursing education 
instructor. These instructors were selected because of the 
number of students in their programs that have transitioned 
through B.E. preparation courses. 
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Monitoring & Screening Criteria and 

Data Element Definition/Measure Benchmark Review Level 



Criterion Is Documented evidence that program accomplishes its purpose. 



GED/HSED: Completion of 
program. 

Completion of one or 
more GED tests. 


% of students who complete program 
from those who state GED/HSED as 
goal . 




A decline of greater 
than 10% of the last 
three years . 


Academic Support for 
Post -Secondary Programs 
(ASPP) 


% of students who improve basic 
skills in reading, English, or math 
or % of students who complete the 
program course (s) for which they 
are receiving instruction. 




Less than 70% improve 
basic skills by one 
or more grade levels 
in reading, English, 
or math. Or less 
than 67% complete the 
program course (s) for 
which they enrolled. 


Post -Secondary 
Preparation enrollment 
(PSP) : Student enters 
college program 


% of PSP students that enter 
college program within 1 or 2 
semesters . 




Less than 50% of the 
students who enroll 
in PSP courses enter 
college program 
within 1 or 2 
semesters . 


Engl ish-As -A- Second 
Language (ESL) : 

(1) Student meets goal 
for instruction 
(complete GED/HSED, 
enroll in college 
courses, become U.S. 
citizen, etc . ) 

(2) Completion of ESL 
level in which the 
student is currently 
enrolled . 


(1) % of students who accomplish 
stated goals . 

(2) % of students who successfully 
complete* ESL level in which they 
are enrolled 




Less than 70% make 
significant gains 
toward accomplishment 
of stated goals for 
ESL level in which 
they are enrolled. 


Work Place Literacy 
(WPL) : On-the-job 
instruction in B.E. 
courses leading to 
improved B.E. skills. 


% of students who successfully 
complete* course in which they are 
enrolled (See Issue 2 on last 
page) 




Less than 70% 
complete the course 
in which they are 
enrolled . 


High School Completion 
(HSC) : Adu Its and 
Alternative High School 
students enrolled in 
B.E. courses for 
transfer to High School 
credit . 


% of students who successfully 
complete* course in which they are 
enrolled . 




Less than 70% 
complete the course 
in which they are 
enrolled . 



(appendix continues) 
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Criterion 2t Documented evidence that program prepares students adequately for 
entry level employment or transfer. 


[GED/HSED] 

1) Achievement of 
core 

abilities 

2) Graduate 
satisfaction 

3) % entering 
Nicolet/ 
another college. 






Less than 90% 
satisfaction rate. 
Less than 30% attend 
college within three 
years of completion. 


[ASPP] 

1) Student 
satisfaction 

2) Achievement of 
core abilities. 


(2) Will be developed in 
connection with assessment. 




Less than 67% student 
satisfaction rate 


[PSP] Documented 
evidence that the PSP 
courses prepare 
students adequately 
for program courses 
at NATO or another 
college . 


% of PSP course completers who 
successfully complete their 
first semester of college 
program course work at NATC or 
another college. 




Less than 70% 
successfully complete 
their first semester 
of college program 
course work at NATC 
or another college. 


English -As -A- Second 
Language (ESL) : 

(1) Achievement of 
core abilities. 

(2) Graduate 
satisfaction . 


(1) Will be developed in 
connection with assessment. 

(2) Percentage of graduates who 
report that they are very 
satisfied, or satisfied, with 
the training they received on a 
six month follow-up survey. 




Less than 90% 
satisfaction rate. 


(WPL) : Course 

completer 

satisfaction 


(1) Will be developed in 
connection with assessment. 

(2) Percentage of graduates who 
report that they are very- 
satisfied, or satisfied, with 
the training they received on a 
six month follow-up survey. 




Less than 90% 
satisfaction rate. 


(HSC) : Course 
completer 
satisfaction 


(1) Will be developed in 
connection with assessment. 

(2) Percentage of graduates who 
report that they are very 
satisfied, or satisfied, with 
the training they received on a 
six month follow-up survey. 




Less than 90% 
satisfaction rate. 



(appendix continues) 
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Criterion 3i Documented evidence of student demand for pr 


ogram' s curriculum. 


[GED/HSED] 

Course Utilization 


(1) unduplicated number 
of enrolled students, 
June 1 - July 31. (2) 

Number of FTE generated 
in instructional area 
courses June 1 - May 31. 




A decline of 
greater than 10% 
over the past 
three years . 


[ASPP] 

Course Utilization 


(1) unduplicated number 
of enrolled students, 
July 1 - June 30. (2) 

Number of FTE generated 
in instructional area 
courses in fiscal year 
(June 1 - May 31) . 




A decline of 
greater than 10% 
over the past 
three years . 


[PSP] 

Course utilization 


(1) Unduplicated number 
of paid students enrolled 
in PSP courses , June 1 - 
May 30. (2) Number of FTE 

generated in fiscal year. 


Pattern 

of 

increase 
over past 
three 
years 


A decline of 
greater than 10% 
over the past 
three years . 


(ESL) : 
Course 
utilization . 


(1) unduplicated number 
of enrolled students, 
June 1 - July 31. (2) 

Number of FTE generated 
in instructional area 
courses June 1 - May 31. 




A decline of 
greater than 10% 
over the past 
three years . 


(WPL) : 
Course 
utilization 


(1) unduplicated number 
of enrolled students, 
June 1 - July 31. (2) 

Number of FTE generated 
in instructional area 
courses June 1 - May 31. 




A decline of 
greater than 10% 
over the past 
three years . 


(HSC) : 

Course utilization 


(1) unduplicated number 
of enrolled students, 
June 1 - July 31. (2) 

Number of FTE generated 
in instructional area 
courses June 1 - May 31. 




A decline of 
greater than 10% 
over the past 
three years . 



(appendix continues) 




53 



53 



Page 4 



Criterion 4: Documented evidence of student retention. 



[GED/HSED] 

Course completion 



Persistence 



% of students officially 
enrolled in a course on date of 
record who successfully complete 
the course (i.e. get a passing 
grade) , aggregated for all 
courses in the instructional 
area, term-by-term. 

% of noncompleters who re-enroll 
without interruption 



[ASPP] 

Course completion 



[PSP] Documented 
evidence of student 
retention 



(ESL) : 

Course Completion 



{ WPL) : 

Course completion 



(HSC) : 

Course completion 



% of students officially 
enrolled in a course on date of 
record who successfully complete 
the course (i.e. get a passing 
grade) , aggregated for all 
courses in the instructional 
area, term-by-term. 

% of students officially 
enrolled in prep courses on date 
of record who successfully 
complete their courses, as 
defined by getting a passing 
grade . 

% of students officially 
enrolled in an ESL course on 
date of record who successfully 
complete the course (i.e. get a 
passing grade) , aggregated for 
all courses in the instructional 
area, term-by-term. 



% of students officially 
enrolled in a course on date of 
record who successfully complete 
the course (i.e. get a passing 
grade) , aggregated for all 
courses in the instructional 
area, term-by-term. 



% of students officially 
enrolled in a course on date of 
record who successfully complete 
the course (i.e. get a passing 
grade) , aggregated for all 
courses in the instructional 
area, term-by-term. 



An average of less 
than 70% over the 
last three years. 



Less than 60% 
return the next 
semester . 



An average of less 
than 80% over the 
last three years. 



Less than 65% 
complete their 
courses with a 
passing grade 



Less than 70% 
complete the ESL 
level in which they 
are currently 
enrolled . 



Less than 65% 
complete their 
courses with a 
passing grade 



Less than 65% 
complete their 
courses with a 
passing grade. 
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Criterion 5s Documented evidence of cost efficiency in delivery of 
instruction. 


[GED/HSED] : 

FTE per instructor 


Student FTE in instructional 
area courses divided by total 
instructor workload in those 
courses/ from June 1 through 
May 31. 




Less than 8 FTE 
/instructor for 
all lab 
instruction 
(includes cost of 
ESL/ HSED/GED/ 
Remedial / etc . ) . 


[ASPP] : 

FTE per instructor 


Student FTE in instructional 
area courses divided by total 
instructor workload in those 
courses/ from June 1 through 
May 31. 




Less than 8 FTE 
/instructor for 
all lab 
instruction 
(includes cost of 
ESL/ HSED/GED/ 
Remedial , etc . ) . 


[PSP] : 

FTE per instructor 


Student FTE in instructional 
area courses divided by total 
instructor workload in those 
courses/ from June 1 through 
May 31. 




Less than 8 FTE 
/instructor for 
all lab 
instruction 
(includes cost of 
ESL, HSED/GED/ 
Remedial / etc . ) . 


(ESL) : 

FTE per instructor 


Student FTE in instructional 
area courses divided by total 
instructor workload in those 
courses/ from June 1 through 
May 31. 




Less than 8 FTE 
/instructor for 
lab instruction ( 
includes cost of 
ESL, HSED/GED/ 
Remedial / etc . ) . 


( WPL) : 

FTE per instructor 


Student FTE in instructional 
area courses divided by total 
instructor workload in those 
courses/ from June 1 through 
May 31. 




Less than 8 FTE 
/instructor for 
lab instruction ( 
includes cost of 
ESL, HSED/GED, 
Remedial , etc . ) . 


(HSC) : 

FTE per instructor 


Student FTE in instructional 
area courses divided by total 
instructor workload in those 
courses/ from June 1 through 
May 31. 




Less than 8 FTE 
/instructor for 
lab instruction ( 
includes cost of 
ESL, HSED/GED, 
Remedial, etc . ) . 
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Criterion 6* Multiple options for access to program's curriculum. 



[GED/HSED] 
Access Options 


Number of alternative access 
options available, in 
structure, location, time, 
work-connection, and/or 
outcome . 




Decline in number 
of access options 
available . 


[ASPP] 

Access Options 


Number of alternative access 
options available, in 
structure, location, time, 
work-connection, and/or 
outcome . 




Decline in number 
of access options 
available . 


[PSP] : 

Access Options 


Number of alternative access 
options available, in 
structure, location, time, 
work-connection, and/or 
outcome . 




Decline in # of 
access options 
available 


[ESL] : 

Access Options 


Number of alternative access 
options available, in 
structure, location, time, 
work-connection, and/or 
outcome . 




Decline in # of 
access options 
available 


(WPL) : 

Access Options 


Number of alternative access 
options available, in 
structure, location, time, 
work-connection, and/or 
outcome . 




Decline in # of 
access options 
available 


(HSC) : 

Access Options 


Number of alternative access 
options available, in 
structure, location, time, 
work-connection, and/or 
outcome . 




Decline in # of 
access options 
available 


Other 


Number of alternative access 
options available, in 
structure, location, time, 
work-connection, and/or 
outcome . 




Decline in # of 
access options 
available 



Issues: (1) Need to track PSP students who transfer to other college, the 



semester after completing PSP/GED/HSED course or program. 

(2) Need to develop tracking data for number of WPL students who 
complete course and apply knowledge/certificate to employment 
needs . 

(3) Need to develop a system to gather information on student 
satisfaction. 

♦Definition of Successful Completion: % of students officially enrolled on 

date of record who successfully complete the course (i.e., get a passing 
grade) : Aggregated for all courses in the instructional area. 
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Appendix D 

Program Evaluation Instrument 

PROGRAM EVALUATION AND IMPROVEMENT RECOMMENDATION FORM 

PAGE 1 

PROGRAM TITLE: BASIC EDUCATION 
DATES OF PROGRAM EVALUATION: 

INTERNAL/ FORMATIVE PROGRAM REVIEW TEAM: 

(CHAIR) 

(CO-Chair) 



The Program Evaluation Review Team for Basic Education consists 
of four full-time faculty in the B.E. program; two members of 
the B.E. Advisory Committee; one academic advisor; and two non- 
B.E. instructors. 



DATE OF COMPLETED REPORT: 
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PAGE 2 

DIRECTIONS: This form is intended to capture the Program Review 
Team's analysis, insights, comments, and recommendations about 
the educational offerings of the Basic Education Programs. There 
are five parts plus a section for recommendations. Specific 
directions are provided for each part. 

(All recommendations and comments from this program evaluation 
will be considered by the Teaching, Curriculum, and Instructional 
Resources standing committee (TCIR) , as it prioritizes budget, 
curriculum, programmatic staffing, and other issues for the next 
academic year.) 



EVALUATION QUESTIONS AND TOPICS 
FOR CONSIDERATION 


ANALYSIS, INSIGHTS, COMMENTS 


ACTION 

INDICATED 


I. PROGRESS REPORT 

Please review the action plan resulting from this year's evaluation. For 
each item, indicate whether it has been recommended for completion or drop. 


II. CURRENT EFFECTIVENESS AND 
EFFICIENCY 

Do any of the monitoring and 
screening data elements fall 
below the proposed review 
level? If so, identify issues 
and possible actions to address 
that criterion or data element. 
Otherwise, analyze trends or 
themes from the monitoring and 
screening data, or move 
directly to Section IIA. 






IIA. Are there multiple options 
for success? 


Check the options available 
in program courses . 

I TV Evening 

VIDEO Weekend 

Indiv. Arr. Open Entry/ 

Internet Open Exit 

Other 





(appendix continues) 




58 



58 



Page 3 



EVALUATION QUESTIONS AND ANALYSIS, 

TOPICS FOR CONSIDERATION COMMENTS 



III. TRENDS AFFECTING THE 
B.E. PROGRAMS 



INSIGHTS, 



ACTION 



INDICATED 



Consider the trends 
information provided in the 
research; consult with B.E. 
Advisory Committee; review 
course and program outcomes; 
and assessment data to 
answer the following 
questions : 

A. What are the emerging 
trends in B.E. and the 
occupations/further 
education B.E. supports? 

B. What types of education, 
knowledge, perspectives, 
skills, and sensibilities 
will be needed in the near 
future to be successful in 
B.E. ? 



C. How well does the 
current B.E. curriculum and 
student learning address the 
listed trends and needs? 

What changes in the content 
and/or delivery should be 
made to better prepare 
students for the demands 
they will face upon 
completion of B.E. at NATC? 
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EVALUATION QUESTIONS AND 
TOPICS FOR CONSIDERATION 


ANALYS IS , INS IGHTS , 
COMMENTS 


ACTION 

INDICATED 


IV. ASSESSMENT 

What assessment loops are 
currently used? [An 
assessment loop = 

(1) identification of desired 
learning outcomes; 

(2) assessment of students' 
learning or those outcomes, 
and (3) modification of 
teaching content/strategies 
to improve learning of those 
outcomes] . 

A. For individual B.E. 
courses? 

B. For B.E. program's 
technical skills? 

C. For core abilities? 

For any of the above in 
which the assessment loop is 
not yet completely in place, 
identify actions needed to 
take during the next year(s) 
to complete the process. 
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EVALUATION QUESTIONS AND TOPICS 
FOR CONSIDERATION 


ANALYSIS , INSIGHTS , 
COMMENTS 


V. INSTRUCTIONAL TECHNOLOGY 




A. Identify current instructional 
technology within the B.E. 
programs (or discipline) labs and 
classrooms . 




(Please use the provided 
technology inventory list to 
assist with this task.) 




B. Prepare a two-year plan for 
improvement in instructional 
technology, for all Basic 
Education (B.E.) Programs, with 
rationale supporting the need and 
with projected costs where 
possible 




[refer to attached cost allocation 
budget form] . 




C. Identify staff development 
needed to implement the planned 
technology effectively. 




D. Identify any issues or 
barriers related to the planned 
technology improvement. 





ACTION 

INDICATED 
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EVALUATION QUESTIONS AND TOPICS 
FOR CONSIDERATION 


ANALYSIS , INSIGHTS, 
COMMENTS 


ACTION 

INDICATED 


Check Point: 

Does the committee have sufficient 
information to make 
recommendations? 

If so, continue with VI. 
Prioritized Recommendations . 

VI. PRIORITIZED RECOMMENDATIONS 

Recommendations should contribute 
to the improvements indicated in 
the analysis, insights, and 
comments under Sections II, III, 
and IV. The supporting statement 
should show which issue the 
recommendation is intended to 
improve and how/why it is expected 
to do so. 

A. List recommendations that can 
be implemented by program faculty 
working as a self-contained unit 
(Basic Education) . Include plan 
of action for each of the 
recommendations . 

List each recommendation in 
committee's priority order. 

B. List recommendations that 
require institutional support or 
collaboration with staff outside 
of B.E. (List them in prioritized 
order. ) 
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Basic Education 
Nicolet Area Technical College 
REQUEST FOR NEW ALLOCATION 
1999-2000 Budget Development Process 



1. Organizational 
Area, Contact 
Person, and Phone 
Number : 


Basic 365-4492 

Education 


2 . Type of 
Request : 

(Check one: State 
total amount 
requested; 
attach itemized 
list) . 


Base 

Allocation 

increase 


New Staff 


One-time 

Operating 

Cost 


Capital 

Expenditure 


3. Rationale for Requests Explain how the funds will be used and why they 
are needed. Where appropriate, show relationship between this request and 
monitoring/screening data, trends data, assessment data, and program 
evaluation recommendations. Indicate how these expenditures will benefit 
teaching and learning and/or support Nicolet' s strategic goals and what the 
impact will be if the funds are not approved. 




4. Sources Indicate how the needed 
dollar amount was determined; attach 
quotes for capital items. 




5. Time Lines when (mo & yr) must 
this item be available to meet the 
needs indicated above? 




6. Long-Term Financial Implications s 
Describe any long-term implications 
for the budget (e.g., increased 
revenue, ongoing financial support, 
etc • ) . 





O 

ERIC 
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Appendix E 
Summative Committee 

♦ The B.E. cluster coordinator served because of the 
affiliation with all components of the B.E. program. 

♦ Two B.E. Advisory Committee members: (a) one consisted of an 

employee from a local factory, representing business and 
industry, and (b) one member represented the area job 
center . 

♦ The vice-president of instruction served, because of the 
over-all supervisory role to B.E. 

♦ A member of the Teaching, Curriculum, and Instructional 
Resources TCIR) standing committee was selected because of 
TCIR' s role in approving all program development projects. 



O 
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Strategic Plan for NATC and Basic Education 
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Strategic Plan One: Excellence in Educational Offerings and 
Delivery. To implement effective and efficient systems and 
structures that ensure continual dynamic alignment and excellence 
of educational offerings with District needs, that provide 
continual improvement of teaching and learning, and that make 
Nicolet the college of first choice for its communities. 

Strategic Plan Two: Growth Through Enrollment Management and 
Image Enhancement. To implement systems for increasing 
enrollments and FTEs, building mutually beneficial partnerships 
and collaborative projects, improving retention and satisfaction 
of students, and enhancing institutional image. 

Strategic Plan Three: Expansion of Collaborative, High-Leverage 
Partnerships. To develop, maintain, and expand high- leverage 
partnerships and collaborative projects that increase FTEs and 
revenues, that integrate technology, and that enhance the image 
of the college and the development of its communities. 

Strategic Plan Four: Quality, Efficiency, and Culture of the 
College: To systematically improve the college's efficiency, 
effectiveness, and accountability through stewardship of 
resources, implementation of continual improvement systems, and 
commitment to a culture of service to students, staff, and the 
community. 
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